Skip to content

[DO NOT MERGE] feature branch for arrow support in bq jdbc - #15067

Draft
keshavdandeva wants to merge 5 commits into
mainfrom
jdbc/arrow-support-feature
Draft

keshavdandeva wants to merge 5 commits into
mainfrom
jdbc/arrow-support-feature

Conversation

@keshavdandeva

Copy link
Copy Markdown
Contributor

No description provided.

#14964)

b/582677347

This PR adds the `EnableQueryResultsArrowFormat` connection property
(default `false`, accepts 0/1/true/false).

It goes through the usual path: URL -> DataSource -> BigQueryConnection
-> BigQuerySettings for each statement, same as `EnableTimestampPicos`.

The property doesn't change any behaviour yet. Reading it and asking
BigQuery for Arrow results comes in a follow-up PR (b/582676785). This
PR only sets up the configuration so that work stays small.

Added unit tests
)

b/582676539

Prepares the Arrow reader for the job stream (b/581544197).

- Producer skips Read API responses without rows, fails fast when a
response reports rows but has no record batch, and forwards a schema
sent in the stream.
- Result set skips zero-row batches and builds its decoder from the
first schema it sees.
- `isBeforeFirst()`/`isLast()` return false for an empty result.
b/582677690

Adds a reader for a query job's `_default` stream, so results can be
read through the Storage Read API without creating a read session.
Nothing calls it yet; Task 4 (b/582676785) wires it into statement
execution.

## Changes

- Build the stream name from the job reference (project, location, job
ID).
- Send the Arrow options on every `ReadRows` request. A resumed request
is the first request with only the offset changed.
- The reader forwards each message's `row_count` and decides nothing.
`BigQueryArrowResultSet.loadNextBatch()` fails when a batch doesn't hold
the rows its message reports.
- Any stream error ends iteration: the next call to `next()` returns
false, and getters throw `SQLException` until then.

## Behaviour changes on the existing HTAPI path

- A `row_count` that differs from the rows in the batch is now an error.
- After a stream error, `next()` returns false. Before, it read on from
the following message.

## Testing

- `make lint` and `make unittest`: 1280 tests, 0 failures, 1 skipped in
a class this PR doesn't touch.
- 16 new unit tests. The result-set tests use real Arrow batches.
@keshavdandeva keshavdandeva added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Nov 18, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the EnableQueryResultsArrowFormat configuration to support reading query results in Apache Arrow format. It updates BigQueryArrowBatchWrapper to include schema and row count details, refactors BigQueryArrowResultSet to dynamically load batches and validate row counts, and adds corresponding unit tests. The review feedback highlights a compilation error in BigQueryArrowResultSet due to an unhandled InterruptedException in loadNextBatch(), an off-heap memory leak in the test helper idRoot from unclosed Arrow resources, and improper interruption handling during stream retries in BigQueryStatement.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Indicates a pull request not ready for merge, due to either quality or timing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant